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SUMMARY  PAGE 


THE  PROBLEM 

In  many  instances,  small  sets  of  items  to  be  represented  on 
visual  displays  can  be  depicted  by  single  words  or  simple 
pictures.  Studies  were  conducted  to  determine  if  one  kind  of 
representation  or  the  other  can  be  more  efficiently  processed, 
leading  to  overall  better  operator  performance. 

THE  FINDINGS 

The  pictures  and  words  tested  were  visually  recognized  with 
equal  facility.  However,  when  decisions  involving  semantic 
interpretation  of  written  representations  were  required,  responses 
were  faster  when  those  representations  were  simple  pictures  than 
when  they  were  single  words.  This  picture  advantage  was  found  even 
for  a  task  that  was  hypothesized  to  be  'verbal’  in  nature  and  was 
expected  to  lead  to  a  word  advantage.  It  is  hypothesized  that 
these  response  time  differences  reflect  underlying  differences  in 
the  cognitive  processing  of  the  two  types  of  representations. 

APPLICATION 

These  findings  suggest  that  for  small  sets  of  concrete  items, 
it  is  better  to  use  pictures  than  words  to  represent  them  on 
visual  displays.  This  conclusion  appears  valid  regardless  of  the 
task  involved  or  how  other  information  on  the  display  is 
represented. 
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ABSTRACT 


Previous  research  has  shown  that  for  several  different  kinds  of 
tasks,  decisions  requiring  semantic  interpretation  of  written 
representations  can  be  made  faster  when  those  representations  are 
simple  pictures  than  when  they  are  single  words  depicting  the  same 
items.  These  response  time  differences  are  thought  to  reflect 
underlying  differences  in  the  cognitive  processing  of  the  two  types 
of  representations.  Despite  the  empirical  evidence,  there  is  reason 
to  believe  that  the  response  time  advantage  for  interpretation  of 
pictures  could  be  negated  or  reversed,  depending  on  the  specific  set 
of  pictures  and  words  used  and  on  the  information  requirements  of 
the  particular  task  used.  If  the  picture  advantage  is  indeed 
stimulus-  or  task-dependent,  it  would  preclude  any  general  rule  of 
thumb  for  choice  of  representation. 

Six  pictures  and  six  words  representing  a  small  set  of  concrete 
items  were  used  in  four  different  kinds  of  tasks  to  investigate  the 
possibility  that  stimulus  and  task  variables  affect  response  times 
to  pictures  and  words.  The  first  task  was  a  recognition  task  which 
tested  whether  perceptual  characteristics  of  pictures  and  words 
accounted  for  response  time  differences.  The  other  three  tasks  were 
memory  tasks,  two  of  which  were  hypothesized  to  lead  to  a  picture 
advantage  and  the  third  of  which  was  hypothesized  to  lead  to  a  word 
advantage  due  to  the  nature  of  task  requirements. 

Results  indicated  that  the  speed  of  recognition  of  the  picture 
and  words  sets  was  comparable,  suggesting  that  perceptual 
differences  between  the  sets  of  pictures  and  words  were  not 
responsible  for  response  time  differences  in  the  memory  tasks.  When 
decisions  involving  semantic  interpretation  of  the  stimuli  were 
required  (the  memory  tasks) ,  responses  were  faster  for  pictures  than 
for  words.  This  picture  advantage  was  found  even  for  the  task  that 
was  hypothesized  to  lead  to  a  word  advantage.  It  is  concluded  that 
stimulus  and  task  variables  minimally  affect  response  time 
differences  between  pictures  and  words.  There  appears  to  be  an 
overall,  robust  processing  advantage  for  pictures  which  is 
independent  of  stimulus  and  task  variables. 
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The  Effect  of  Stimulus  and  Task  Variables  on 
Response  Times  to  Pictures  and  Words 


Visual  display  terminals  (VDT's)  are  one  of  the  most  important 
man-machine  interfaces  in  many  modern  military  combat  systems,  such 
as  sonar  and  fire-control  systems  aboard  submarines.  The  complexity 
of  these  systems,  both  in  terms  of  the  technological  sophistication 
of  the  equipment  involved  and  in  terms  of  the  problem  domain  itself, 
usually  results  in  an  overwhelming  amount  of  information  being 
provided  to  the  system  operator  via  VDT's  and  other  interfaces.  The 
sheer  amount  of  information  that  must  be  processed,  coupled  with 
frequent  time  constraints,  dictate  that  operators  be  able  to  read 
and  interpret  the  visual  symbols  representing  information  on  the 
VDT's  as  efficiently  as  possible. 

There  are  many  different  kinds  of  visual  symbols  used  on  visual 
displays,  including  written  words,  alphabetic  abbreviations, 
numerals,  graphic  symbols,  and  pictographs.  In  some  cases,  the  same 
information  can  be  depicted  by  more  than  one  type  of  symbol  or 
representation,  such  as  a  picture  and  its  written  word  label  to 
represent  an  object  such  as  a  submarine.  If  there  are  differences 
between  different  kinds  of  symbols,  in  how  efficiently  they  can  be 
processed  by  operators,  then  it  makes  good  sense  to  use  those  kinds 
that  are  most  efficiently  processed  if  designers  are  free  to  choose 
among  more  than  one  kind  of  symbol  for  representation  of  certain 
information. 

Much  previous  work  comparing  the  processing  time  required  to 
interpret  pictures  and  words  representing  the  same  small  set  of 
concrete  items  (Paivio,  1975,  1978a;  Pellegrino,  Rosinski,  Chiesi  & 
Siegal,  1977;  Hogaboam  &  Pellegrino,  1978;  Potter  &  Faulconer,  1975; 
teLinde,  1982)  has  shown  that  pictures  can  consistently  be 
interpreted  faster  than  their  word  labels  when  a  decision  based  on 
the  underlying  meaning  of  the  item  represented  must  be  made.  The 
majority  of  tasks  used  in  these  studies  has  required  a  decision 
about  the  category  membership  of  the  items  represented  (Nelson,  Reed 
&  Walling,  1976;  Hogaboam  &  Pellegrino,  1978;  Potter  &  Faulconer, 
1975;  Paivio,  1975;  Pellegrino  et.  al.,  1977;  Snodgrass  &  Asiaghi, 
1977) .  Other  tasks  have  required  judgments  about  the  relative  value 
of  pairs  of  items  on  some  dimension  such  as  size  (Paivio,  1975; 
teLinde,  1982),  monetary  value  (Paivio,  1978a),  or  intelligence 
(Banks  and  Flora,  1977) .  The  empirical  support  for  faster 
interpretation  of  pictures  is  substantial.  In  addition,  there  is  a 
large  number  of  different  kinds  of  tasks  for  which  a  picture 
advantage  exists  (Smith  &  Magee,  1980;  teLinde,  1982;  Kroll  & 

Potter,  1984) .  The  only  tasks  for  which  there  seems  to  be  a  clear 
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advantage  for  words  involve  naming  or  pronunciation  (Fraisse,  1968; 
Paivio,  1975) ,  and  this  can  be  explained  simply  by  the  fact  that 
words  have  faster  access  to  phonological  information  than  pictures. 

Several  other  lines  of  research  have  converged  on  the 
conclusion  that  pictures  can  access  meaning  more  quickly  than  words 
representing  the  same  items,  and/or  items  are  better  remembered  when 
represented  as  pictures  than  when  represented  as  words.  One  line  of 
research  has  involved  the  use  of  priming  paradigms  (Vanderwart, 

1984)  and  interference  paradigms  (Biederman  &  Tsao,  1979;  Dhawan  & 
Pellegrino,  1977;  Babbitt,  1982)  in  conjunction  with  free  recall. 
Other  lines  of  research  have  involved  visual  field  studies  (Hatta, 
1977;  Sasanuma,  Itoh,  Mori  &  Kobayashi,  1977),  and  investigations  of 
aphasic  patients  (e.g.,  Sasanuma,  1974).  Although  the  magnitude  of 
the  picture  advantage  found  varied  considerably  among  studies  and 
tasks,  the  consistent  advantage  of  interpreting  pictures  versus 
words  suggests  that  when  no  other  constraints  dictate  choice  of 
representations  for  a  small  set  of  concrete  items,  pictures  should 
be  used  instead  of  words. 

There  are  at  least  two  problems  with  this  conclusion  in  light* 
of  the  studies  to  date,  however.  First,  in  most  of  the  studies  which 
investigated  picture-word  response  time  differences,  with  the 
exception  of  Fraisse  (1968)  and  teLinde  (1982) ,  the  picture  and  word 
sets  were  not  compared  for  perceptual  recognition  time.  If 
perceptual  processes  account  for  interpretational  differences 
between  pictures  and  words,  it  is  much  more  likely  that  these 
differences  are  dependent  on  the  particular  picture  and  word  sets 
tested.  Many  stimulus  attributes,  such  as  visual  extent  or  size, 
discriminability ,  confusability,  familiarity,  etc.,  can  affect  how 
quickly  stimuli  are  perceptually  processed.  If  picture  advantages 
are  stimulus  set-specific,  then  a  general  rule  of  thumb  for  any  set 
of  pictures  is  inappropriate. 

Second,  one  of  the  current  psychological  theories  of  memory 
processing  that  has  been  used  to  explain  picture-word  response  time 
differences,  dual-coding  theory  (Paivio,  1975) ,  suggests  that  words 
should  be  processed  faster  than  pictures  for  a  certain  class  of 
tasks.  Dual-coding  theory  assumes  the  existence  of  two  separate 
long-term  memory  (LTM)  systems,  one  designed  for  the  coding, 
storage,  manipulation,  and  retrieval  of  verbal  or  language  related 
information  and  the  other  designed  for  the  coding,  storage, 
manipulation,  and  retrieval  of  visual  or  spatial  information. 

Picture  stimuli  have  faster  access  to  the  visual  LTM  and  word 
stimuli  have  faster  access  to  the  verbal  LTM.  Paivio  (1975,  1978b) 
has  suggested  that  the  variety  of  tasks  that  have  been  used  to  date 
to  compare  the  processing  of  pictures  and  words  have  been  heavily 
weighted  toward  information  that  would  be  likely  be  stored  in  the 


visual  LTM,  and  hence  pictures  have  been  generally  processed  faster. 
Paivio's  theory  predicts,  however,  that  if  tasks  require  information 
which  is  more  verbally-oriented,  then  words  should  access  that 
information  more  efficiently  than  pictures.  In  practical  terms,  this 
would  mean  that  the  rule  of  thumb  for  designers  to  use  pictures  to 
represent  small  sets  of  concrete  items  on  visual  displays  would  be 
conditional,  dependent  on  the  kind  of  information  required  for  the 
task. 


A  series  of  experiments  was  formulated  to  address  the  issues  of 
-the  possible  dependence  of  the  efficiency  of  interpretation  of 
pictures  and  words  on  the  particular  stimulus  sets  used  and  on  the 
kind  of  information  required  for  the  task.  First  a  recognition  task 
was  used  to  evaluate  whether  visual  recognition  times  were 
comparable  for  the  sets  of  pictures  and  words  used.  Additionally, 
performance  in  three  different  tasks  using  the  same  picture  and  word 
sets,  but  differing  in  the  kind  of  information  required,  was 
studied.  The  profile  of  picture-word  response  times  across  tasks 
wills  1)  determine  the  contribution  of  recognition  differences 
between  pictures  and  words  to  response  time  differences  in  memory 
tasks;  and  2)  test  the  prediction  of  the  dual-coding  hypothesis  that 
the  picture  advantage  is  task-dependent.  On  the  practical  level, 
this  study  will  determine  whether  the  rule  of  thumb,  use  pictures  to 
represent  small  sets  of  concrete  items,  is  generally  applicable 
across  stimulus  sets  and  tasks. 


Experiment  1 

In  this  experiment  subjects  had  to  determine  whether  or  not  a 
picture  or  word  test  stimulus  represented  the  same  item  as  a  picture 
or  word  cue.  It  was  designed  to  determine  if  the  picture  and  word 
sets  used  for  the  subsequent  experiments  were  comparable  in  terms  of 
recognition  speed. 

Method 


Subjects.  Twenty-eight  males  ranging  in  age  from  18  to  42,  with 
a  median  age  of  19,  volunteered  to  participate  in  Experiment  la. 
Nineteen  were  students  entering  basic  submarine  school  at  the 
Subase,  Groton,  CT.,  and  nine  were  staff  members  at  the  Naval 
Submarine  Medical  Research  Laboratory,  Subase,  Groton,  CT.  Fourteen 
male  staff  members  at  the  Naval  Submarine  Medical  Research 
Laboratory  volunteered  to  participate  in  a  second  part  of  this 
experiment  (lb).  These  subjects  ranged  in  age  from  20  to  43,  with  a 
median  age  of  26.  Volunteers  were  naive  as  to  the  specific 
hypotheses  being  investigated,  but  were  told,  in  general,  that  the 
purpose  of  the  tests  was  to  determine  how  people  mentally  process 
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pictures  and  words  under  various  conditions. 


Stimuli.  The  stimuli  for  this  experiment  were  six  words  and  six 
pictures  (see  Figure  1)  whose  referents  are  real  world  objects  that 
are  commonly  detected  by  sonar  or  radar  on  a  submarine  or  ship.  The 
pictures  were  simple  computer  generated  outline  drawings. 

All  stimuli  subtended  a  maximum  of  4  degrees  visual  angle 
horizontally  and  1.6  degrees  vertically  from  a  viewing  distance  of 
43cm.  The  shorter  words,  such  as  SUB,  SHIP  and  MINE  subtended  a 
smaller  visual  angle  horizontally  than  did  the  longer  words,  as  did 
the  picture  of  the  mine. 

Apparatus .  The  displays  were  presented  on  a  Tektronix  4010 
display  terminal  driven  by  a  Data  General  NOVA  1220  minicomputer. 
Responses  were  made  by  pushing  keys  at  the  ends  of  the  bottom  row  of 
the  terminal's  keyboard,  that  is,  the  ‘z*  and  ’/'  keys.  The  keys 
were  labelled  'yes'  and  'no',  counterbalanced  so  that  half  the 
subjects  had  the  'yes'  key' on  the  left  and  half  had  it  on  the  right. 
The  index  fingers  of  both  hands  were  used  to  respond.  Response  times 
were  measured  internally  by  the  computer  via  a  real  time  clock  which 
was  started  after  the  stimuli  were  drawn  and  stopped  upon  a  key 
press.  The  stimuli  were  completely  drawn  within  30  msec  and  appeared 
to  be  presented  instantaneously.  Responses  and  response  times  were 
recorded  by  the  computer. 

Task.  For  the  recognition  task,  subjects  were  presented  with  a 
cue  in  the  upper  left  corner  of  the  CRT.  The  six  words  of  the 
stimulus  set  were  used  as  cues  for  Experiment  la,  and  the  six 
pictures  of  the  stimulus  set  were  used  as  cues  for  Experiment  lb. 

One  second  after  presentation  of  the  cue,  a  picture  or  word  test 
stimulus  was  presented  in  the  center  of  the  screen.  Both  the  cue  and 
the  test  stimulus  remained  visible  until  a  response  was  made.  Both 
Experiments  la  and  lb  consisted  of  two  blocks  of  36  trials.  For  one 
block  of  trials  the  test  stimuli  were  always  pictures  and  for  a 
second  block  of  trials  the  test  stimuli  were  always  words.  The 
subject's  task  was  to  determine  if  the  stimulus  in  the  center  of  the 
screen  represented  the  same  referent  as  the  cue  item.  For  example, 
if  the  cue  was  a  picture  of  the  'MINE',  and  the  test  stimulus  was 
the  picture  or  the  word  'MINE',  the  subject  pushed  a  key  marked 
'yes',  and  if  the  cue  was  the  word  'TORPEDO',  and  the  test  stimulus 
was  either  the  picture  or  word  'PLANE',  he  pushed  a  key  marked  'no1. 

Within  a  block  of  trials,  each  of  the  six  cues  was  paired  with 
a  matching  test  stimulus  three  times  and  three  randomly  chosen 
non-matching  test  stimuli  once  each.  This  resulted  in  18  correct 
'yes'  and  18  correct  'no'  responses.  The  order  of  presentation  of 
trials  within  a  block  was  randomized  each  time  the  task  was 


administered.  The  order  of  blocks  (picture  or  word  test  stimuli)  was 
counterbalanced  for  practice  and  test  sessions,  such  that  half  the 
subjects  performed  the  picture  test  stimulus  block  first  and  half 
the  subjects  performed  the  word  test  stimulus  block  first  in  each 
session.  There  were  three  practice  sessions  and  one  test  session. 
Experiment  lb  was  administered  after  completion  of  Experiment  la. 

Procedure .  At  the  beginning  of  the  first  practice  session, 
subjects  were  given  written  instructions  describing  the  general 
nature  of  the  experiment  and  the  stimuli  and  specific  instructions 
for  performing  each  task.  While  it  was  made  clear  that  these  were 
response  time  tasks,  it  was  stressed  that  accuracy  should  not  be 
sacrificed  for  speed.  Any  questions  were  answered  at  any. time  during 
the  experiment  at  subjects’  requests.  Practice  and  test  of  the 
tasks  included' in  the  different  experiments  were  administered 
concurrently  to  the  subjects.  The  order  of  practice  on  the  different 
tasks  was  balanced  across  subjects.  All  tasks  were  practiced  three 
times  before  the  test  session  for  that  task.  Pilot  data  had 
indicated  that  asymptotic  response  times  on  these  tasks  were 
achieved  rapidly.  If  subjects  made  more  than  10%  errors  on  any 
practice  session,  they  were  reminded  not  to  sacrifice  accuracy  for 
speed. 

Practice  and  tests  were  performed  in  a  quiet,  darkened  room. 

The  sessions  were  self-paced  in  that  subjects  pressed  a  key  on  the 
keyboard  to  begin  each  trial.  After  each  response,  the  subject's 
response  time,  response,  and  whether  the  response  was  correct  or 
not,  were  displayed  on  the  VDT  screen.  Trials  on  which  errors  were 
made  were  rerun  at  the  end  of  a  block  of  trials  so  that  response 
times  for  correct  responses  were  obtained  for  all  trials.  Each  day's 
practice  required  about  two  hours  per  subject.  On  test  days,  the 
order  of  task  presentation  was  again  balanced  across  subjects.  The 
test  day  consisted  of  identical  warm-up  and  test  sessions  for  the 
recognition  task. 

Results  and  Discussion 


Analyses  of  variance  (ANOVA's)  computed  on  response  times  for 
correct  responses  for  28  subjects  were  used  to  analyze  all  data, 
unless  otherwise  specified.  Both  multivariate  and  univariate  test 
statistics  were  computed  for  each  analysis  *.  Error  rates  were  less 
than  10%  for  all  experiments  reported  here  and  were  not 
statistically  analyzed.  Inspection  of  errors  indicated  that  they 
generally  supported  the  response  time  data,  that  is,  conditions 
leading  to  longer  response  times  also  generally  led  to  more  errors. 

Three-way  repeated  measures  ANOVA's  (Stimulus  Type  (ST)  x  Cue 
Item  (ITEM)  x  Response  (RESP) )  were  computed  on  response  times  for 


the  recognition  task  with  word  cues  (Experiment  la)  for  the  first 
day's  practice  and  the  final  test  session.  The  main  intent  of 
analyzing  the  practice  data  was  to  determine  if  there  was  a 
picture-word  difference  on  initial  presentation  of  the  stimuli. 

There  was  a  main  effect  of  stimulus  type  for  the  first  practice 
(pc. 001),  with  responses  to  the  words  being  faster  than  to  the 
pictures  (mean  response  times  of  417  msec  and  481  msec, 
respectively) .  While  most  other  effects  analyzed  were  also 
significant  for  the  practice  data,  they  did  not  appear  to  confound 
the  picture-word  differences,  and  are  not  addressed  here. 

There  was  a  main  effect  of  stimulus  type  for  the  test  session 
(pc. 05),  words  being  responded  to  faster  than  the  pictures  (mean 
response  times  of  357  msec  and  371  msec,  respectively) .  'Yes' 
responses  (339  msec)  were  significantly  faster  (pc.001)  than  'no* 
responses  (391  msec) .  There  were  also  significant  differences 
(pc. 001)  among  responses  to  the  six  cue  items.  There  were  also 
significant  Stimulus  Type  x  Cue  Item  and  Cue  Item  x  Response 
interactions  found  for  the  test  session  response  times.  Tests  of 
simple  effects  were  computed  to  elucidate  these  interactions.  Simple 
effects  of  stimulus  type  for  each  cue  item  showed  that  pictures  were 
responded  to  significantly  faster  than  words  for  'mine'  (pc.05), 
words  were  responded  to  significantly  faster  than  pictures  for 
'plane '  (pc. 05) ,  'ship'  (pc.05),  and  'sub'  (pc.001),  and  there  were 
no  significant  picture-word  response  time  differences  for  'missile' 
and  'torpedo'.  The  mean  response  times  for  pictures  and  words  for 
each  cue  item  are  shown  in  Figure  2.  Simple  effects  of  response  for 
each  item  showed  that  'yes'  responses  were  significantly  faster  than 
'no'  responses  for  all  items  (pc. 01),  indicating  the  Cue  Item  x 
Response  interaction  was  due  to  differences  in  the  magnitude  of  the 
'yes'  advantage  among  stimulus  items.  The  mean  response  times  for 
'yes'  and  'no'  responses  for  each  item  are  shown  in  Figure  3. 

Although  I  assumed  that  the  appropriate  perceptual 
representations  were  generated  from  the  word  cues  since  subjects 
knew  what  kind  of  test  stimulus  to  expect,  it  was  noted  that  the 
word  cues  might  bias  response  times  in  favor  of  the  word  test 
stimuli . through  the  mechanism  of  visual  priming,  that  is,  the  word 
test  stimuli  matched  the  cues  visually,  while  the  picture  test 
stimuli  did  not.  One  could  argue  that  the  word  advantage  was  due  to 
this  bias,  and  if  this  bias  did  not  exist,  picture  stimuli  might  be 
responded  to  faster  than  word  stimuli.  To  address  this  argument,  14 
subjects  served  in  Experiment  lb,  for  which  the  task  was  identical 
to  the  recognition  task  already  administered,  except  that  the  cues 
were  pictures  instead  of  words.  The  procedure  was  identical  to  that 
used  in  Experiment  la. 

A  three-way  ANOVA  (ST  x  ITEM  x  RESP)  computed  on  test  session 


response  times  showed  that  'yes'  responses  were  significantly  faster 
than  'no'  responses  (F(l,13)  =  36.69,  p<.001),  and  that  Stimulus 
Type  interacted  with  Item  (F(5,65)  =  2.39,  p<.05).  Overall,  pictures 
and  words  were  responded  to  comparably  (354  msec  and  358  msec, 
respectively) .  Tests  of  simple  effects  computed  on  the  ST  x  ITEM 
interaction  showed  that  the  pictures  were  responded  to  significantly 
faster  than  words  for  'missile'  (pc. 05)  and  words  were  responded  to 
marginally  faster  than  pictures  for  'sub'  (p=.ll).  There  were  no 
differences  between  pictures  and  words  for  the  other  four  items.  The 
mean  response  times  to  pictures  and  words  for  each  item  are  shown  in 
Figure  4. 

As  stated  at  the  outset,  much  previous  work  in  the  area  of 
picture-word  processing  differences  (Faivio,  1975, 1978a, 1978b; 
teLinde,  1982;  Smith  &  Magee,  1980;  Potter  &  Faulconer,  1984)  has 
hypothesized  that  memory  processes  rather  than  perceptual  processes 
are  responsible  for  picture-word  response  time  differences,  but  few 
studies  have  included  appropriate  tests  of  possible  perceptual 
processing  differences  between  pictures  and  words.  The  recognition 
task  included  here  was  a  test  of  whether  there  were  differences 
between  the  picture  and  word  sets  when  the  processing  involved  was 
perceptual  or  'front-end'  in  nature.  The  fact  that  the  picture-word 
response  time  differences  in  the  recognition  tasks  were  small 
(Experiment  la  with  word  cues)  or  negligible  (Experiment  lb  with 
picture  cues)  indicated  that  any  major  picture-word  processing 
response  time  differences  in  the  memory  tasks  using  the  same  picture 
and  word  stimulus  sets  could  not  be  attributed  to  differences  in 
perceptual  attributes  of  the  items  of  the  sets. 

The  fact  that  there  was  a  substantial  advantage  for  the 
processing  of  words  in  the  first  practice  session  of  the  task 
indicates  that  subjects  might  have  entered  the  experiment  with 
greater  familiarity  with  the  word  set  than  with  the  picture  set. 

This  greater  familiarity  might  also  account  for  the  small  but 
significant  advantage  for  words  in  the  test  session  with  word  cues. 
Since  that  initial  word  advantage  was  greatly  reduced  after  exposure 
to  the  stimulus  sets,  it  is  suggested  that  familiarity  with  the 
stimuli  had  a  typical  learning  curve  effect  on  processing  time. 
Although  initially  there  were  different  starting  points  for  pictures 
and  words  on  the  learning  curve,  by  the  time  the  test  session  was 
administered,  familiarity  with  both  the  picture  and  word  sets  was 
such  that  performance  was  near  asymptote,  thus  accounting  for  the 
closing  of  the  picture-word  processing  time  gap. 

That  the  picture-word  response  time  differences  varied  from 
item  to  item  indicates,  at  the  individual  item  level,  that  it  is 
difficult,  if  not  impossible,  to  equate  pictures  and  words  on  all 
those  variables  that  do  affect  recognition,  such  as 
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discriminability ,  visual  extent,  confusability ,  etc.  Even  though  the 
sets  of  pictures  and  words  were  fairly  comparable  in  terms  of 
recognition  time,  all  the  individual  items  were  not.  The  picture  of 
the  mine  might  have  been  faster  than  the  word  'mine'  because  its 
shape  was  so  unique  compared  to  the  shapes  of  the  other  pictures. 

The  words  'sub',  'plane1,  and  'ship'  might  have  been  faster  than 
their  picture  representations  because  the  words  subtended  less 
visual  angle  and  were  less  conf usable.  The  point,  again,  is  that  at 
the  individual  item  level  it  is  likely  that  pictures  and  words 
differ  in  stimulus  characteristics  which  affect  recognition,  but  at 
the  level  of  a  set  of  items,  these  factors  offset  each  other  to 
produce  overall  comparability  in  picture  and  word  recognition. 

The  advantage  of  'yes'  responses  for  this  task  is  consistent 
with  much  previous  work.  This  advantage  is  often  explained  as  due  to 
an  extra  processing  stage  for  'no'  responses:  more  stringently 
verifying  that  an  item  is  not  the  one  cued  for. 

Experiment  2 

This  experiment  involved  a  task  similar  to  categorization  tasks 
that  have  been  used  in  previous  studies.  Since  tasks  of  this  kind 
have  typically  led  to  consistent  response  time  advantages  for 
pictures,  it  was  hypothesized  that  this  task  would  also  lead  to  a 
picture  advantage. 

Method 


Subjects.  The  twenty-eight  males  from  Experiment  la  volunteered 
for  this  experiment  also. 

Stimuli  and  Apparatus.  The  stimuli  for  this  experiment  were  the 
six  words  and  six  pictures  used  in  Experiment  1  (see  Figure  1) . 

Task  and  Procedure.  The  stimulus  items  for  this  experiment  were 
categorized  in  three  different  ways:  by  the  maximum  speed  of  the 
contact  (fast  or  slow),  function  (weapon  or  platform  for  a  weapon), 
and  normal  location  in  relation  to  the  surface  of  the  ocean  (above 
or  below) .  The  stimulus  items  were  assigned  to  the  categories  such 
that  three  of  the  items  belonged  to  each  category.  The  items 
assigned  to  each  category  were:  FAST-  plane,  missile,  torpedo;  SLOW- 
sub,  ship,  mine;  ABOVE-  plane,  missile,  ship;  BELOW-  sub,  torpedo, 
mine;  WEAPON-  mine,  missile,  torpedo;  and  PLATFORM-  sub,  ship, 
plane.  Subjects  were  required  to  learn  this  categorization _ for  this 
task,  but  it  was,  of  course,  consistent  with  any  pre-existing 
knowledge  the  subjects  had  about  these  items. 

One  of  the  category  names,  the  cue,  was  presented  in  the  upper 


left  section  of  the  screen  followed  by  a  question  mark  (e.g.,  FAST?) 
and  one  second  later  one  of  the  test  stimuli  was  presented  in  the 
center  of  the  screen.  The  cue  and  the  test  stimulus  remained  on  the 
screen  until  a  response  was  made.  Subjects  pressed  the  'yes’  key  if 
the  test  stimulus  was  described  by  the  cue  represented  and  pressed 
the  'no'  key  otherwise.  Each  session  of  the  categorization  task 
consisted  of  72  trials,  each  of  the  six  category  cues  being 
presented  on  12  trials.  In  these  twelve  trials  each  of  the  six  test 
items  (three  positive  instances  and  three  negative  instances)  was 
presented  twice,  once  as  a  picture  and  once  as  a  word.  Picture  and 
word  test  stimuli  were  not  blocked  as  in  the  recognition  task,  but 
rather  all  test  stimuli  were  presented  in  a  single  session,  with  a 
different  random  sequence  for  each  of  the  subjects. 

The  sequence  of  events  was  identical  to  that  described  in 
Experiment  1.  The  categorization  task  was  practiced  on  three 
consecutive  days,  followed  by  a  warm-up  and  test  session  on  the  test 
day. 

Results  and  Discussion 

A  three-way  ANOVA  (Stimulus  Type  (ST)  x  Category  (CAT)  x 
Response  (RESP) )  was  computed  on  response  times  from  the  test 
session  for  -the  categorization  task.  Response  times  for  pictures 
(621  msec)  were  faster  than  those  for  words  (664  msec)  (F(l,27)  = 
10.39,  p< .005) .  'Yes'  (599  msec)  responses  were  significantly  faster 
than  'no'  (655  msec)  responses  (F(l,27)  =  28.31,  p<.001).  Mean 
response  times  to  the  six  categories  were  significantly  different 
(Pillais  =  .747,  Approx  F(5,23)  =  13.58,  pc. 001).  This  main  eff-ect 
of  Category  was  further  analyzed  using  Newman-Keuls  tests.  The 
Newman-Keuls  tests  showed  that:  responses  to  the  'slow*  category 
were  significantly  slower  than  responses  to  all  other  categories 
(p< .01) ;  responses  to  the  'below'  category  were  significantly 
slower  than  responses  to  the  'platform'  (p<.01)  and  'weapon'  (pc. 05) 
categories;  and  responses  to  the  'fast'  category  were  significantly 
slower  than  responses  to  the  'platform'  (pc. 01)  category.  The  mean 
response  times  to  the  different  categories  are  depicted  in  Figure  5. 
There  were  no  significant  interactions. 

The  significant  overall  picture  advantage  in  this  task,  43 
msec,  is  consistent  with  previous  research  (e.g.,  Paivio,  1975; 
Pellegrino  et.  al.,  1977),  that  is,  it  falls  in  the  range  of 
response  time  advantages  commonly  reported.  Since  the  picture 
advantage  reported  here  is  not  due  to  how  fast  the  stimulus  items 
were  recognized,  there  must  be  a  difference  between  pictures  and 
words'  in  the  way  they  are  coded  or  processed  subsequent  to  visual 
recognition.  Differences  among  categories  indicate  that  the  ease  of 
determining  category  membership  depends  on  the  category,  regardless 
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of  whether  items  are  presented  as  pictures  or  words. 

Experiment  3 

This  experiment  involved  a  relative  judgment  task,  here  termed 
the  LTM  task.  This  task  was  also  designed  to  be  comparable  to 
relative  judgment  tasks  that  have  been  used  in  previous  studies.  The 
aspect  of  each  item  that  was  judged  in  this  task  was  its  threat 
level.  A  picture  advantage  was  expected  in  this  task  also,  as 
previous  studies  using  similar  tasks  consistently  showed  ro  us  _ 
picture  advantages.  Since  the  same  stimulus  sets  were  used  for  this 
task  as  for  the  categorization  task,  absolute  differences  m  the 
processing  of  pictures  and  words  between  the  two  tasks  can  be 
attributed  to  different  task  demands. 


Method 

Subjects.  The  same  subjects  participated  in  this  experiment  as 
in  Experiment  2 . 

Stimuli  and  Apparatus.  The  stimuli  and  apparatus  for  this  ^ 
experiment  were  the  same  as  used  in  Experiments  1  and  2 .  The  ' z |  and 
'i/i  keys  used  to  respond  were  labeled  'left*  and  'right1  for  this 

task. 

Task  and  Procedure.  For  the  task  used  in  this  experiment,  the 
LTM  task,  the  six  stimulus  items  were  ordered  from  highest  to  lowest 
threat,  based  on  their  general  threat  level  to  a  submarine  under  a 
variety  of  scenarios.  Starting  with  the  highest  threat,  the  order 
was:  TORPEDO,  MISSILE,  SUB,  SHIP,  MINE,  PLANE.  Again,  this  order  was 
generally  compatible  with  subjects'  pre-existing  knowledge  about  the 
items,  as  evidenced  in  the  ease  of  learning. 

On  each  trial,  two  stimuli  were  presented  side  by  side  in  the 
center  of  the  screen,  either  both  pictures  or  both  words.  Subjects 
were  required  to  press  a  key  on  the  left  or  right  of  the  keyboard, 
indicating  which  of  the  two  stimuli  was  the  greater  threat  as 
instructed.  There  were  15  possible  stimulus  pairs,  consisting  of 
each  item  paired  with  every  other  item.  Each  pair  was  presented  with 
each  item  in  both  the  left  and  right  positions  on  different  trials, 
both  as  pictures  and  as  words,  resulting  in  four  presentations  of 
each  pair  and  60  trials  total.  As  with  the  categorization  task  used 
in  Experiment  2,  trials  with  picture  and  word  stimuli  weremixed 
within  the  same  session.  Stimulus  combinations  were  randomized  or 
each  session. 

As  in  Experiment  1,  the  task  was  practiced  on  three  consecutive 
days,  followed  by  a  warm-up  and  test  session  on  the  test  day. 
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Results  and  Discussion 

A  three-way  ANOVA  (Stimulus  Type  (ST)  x  Stimulus  Pair  (PAIR)  x 
Response  (RESP) )  was  computed  on  response  times  from  the  test 
session  for  the  LTM  task.  Pictures  (637  msec)  were  responded  to 
significantly  faster  than  words  (776  msec) (F(l, 27)  =  62.99,  Pc.001), 
•left'  (693  msec)  responses  were  faster  than  'right1  (721  msec) 
responses  (F(l,27)  =  7.35,  pc.Ol),  and  there  were  significant 
response  time  differences  among  the  stimulus  pairs  (Pillais  =  .921, 
Approx.  F ( (14 , 14)  =  11.67,  pc.001  ).  Additionally,  there  was  a 
significant  ST  x  PAIR  interaction  (F(14,378)  =  5.99,  pc. 005). 

Simple  effects  tests  were  computed  to  determine  the  effect  of 
stimulus  type  for  responses  to  each  stimulus  pair.  These  tests 
showed  that  pictures  were  responded  to  significantly  faster  than 
words  (pc.Ol)  for  all  stimulus  pairs  except  five.  Those  five  were: 
torpedo-missile;  torpedo-plane;  torpedo-ship;  torpedo-sub;  and 
sub-missile.  For  those  five  stimulus  pairs  there  was  no  difference 
in  response  times  to  pictures  and  words. 

The  139  msec  picture  advantage  for  the  LTM  task  is  consistent 
with  picture  advantages  found  for  similar  tasks  by  other 
researchers.  Relative  judgments  involving  comparison  of  pairs  of 
pictures  and  words  on  various  attributes  such  as  size,  monetary 
value,  intelligence,  pleasantness,  and  ferocity  of  the  items 
represented  have  shown  picture  advantages  ranging  from  less  than  100 
msec  to  about  800  msec. 

'Left'  responses  were  faster  than  'right'  responses  probably 
because  of  the  practice  of  reading  from  left  to  right:  The  left 
stimulus  was  read  first  and  thus  processing  of  that  stimulus 
proceeded  ahead  of  processing  of  the  right  stimulus.  The  Stimulus 
Type  x  Pair  interaction  which  was  due  to  a  picture  advantage  for  all 
stimulus  pairs  except  five,  apparently  was  linked  to  the  threat 
level  of  the  correct  response.  Four  of  the  five  pairs  for  which 
there  was  not  a  picture  advantage  included  the  'torpedo',  which  was 
the  greatest  threat.  The  lack  of  a  picture-word  difference  for  pairs 
including  the  greatest  threat  might  have  been  due  to  qualitatively 
different  processing  when  the  greatest  threat  was  one  of  the  test 
pair  (e.g.,  a  response  could  be  based  on  an  absolute  judgement 
rather  than  on  a  relative  judgement) ,  or  it  might  have  simply  been 
due  to  the  fact  that  responses  to  the  greatest  threat  were  faster, 
and  the  rapidity  of  the  responses  simply  did  not  allow  picture-word 
differences  to  occur.  The  pairs  of  pictures  for  which  a  picture 
advantage  did  not  occur  might  simply  have  been  more  difficult  to 
differentiate  because  of  their  visual  similarity  (e.g.,  the  picture 
of  the  torpedo  and  sub  are  very  similar  in  appearance) . 
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Experiment  4 


This  experiment  involved  a  relative  judgment  task  similar  to 
the  one  used  in  Experiment  3,  with  the  difference  being  that  this 
task  was  designed  to  require  storage  of  the  information  needed  to 
perform  the  task  in  short-term  memory  (STM)  rather  than  in  long-term 
memory.  The  STM  task  as  a  whole  was  considered  to  be  a 
verbally-oriented  task,  because  of  the  articulatory  mechanism 
commonly  posited  as  the  mechanism  used  to  retain  information  in  STM. 
Hence  it  was  predicted  that  the  picture  advantage  might  be  negated 
or  reversed  in  this  task.  Addition  of  STM  requirements  to  the  task 
was  assumed  to  generally  require  more  verbally-oriented  processing 
than  if  the  task  required  only  LTM  processing. 

There  is  also  evidence  (e.g.,  see  Zhang  &  Simon,  1985)  that 
there  are  separate  visual  and  auditory  short-term  memories,  and  it 
is  assumed  that  pictures  and  words  might  be  differentially  processed 
in  the  two  STM’s.  Hence  several  STM  parameters  were  varied  which 
were  hypothesized  to  change  the  likelihood  that  information  would  be 
stored  in  the  visual  or  auditory  STM.  It  was  assumed  that  pictures 
would  have  faster  access  to  the  visual  STM  and  words  would  have 
faster  access  to  the  auditory  STM.  It  was  predicted,  therefore,  that 
the  combination  of  manipulations  which  led  to  the  greatest 
probability  of  information  being  stored  in  the  auditory  STM  would  be 
more  likely  to  lead  to  a  word  advantage  and  that  the  combination  of 
manipulations  which  led  to  the  greatest  probability  of  information 
being  stored  in  the  visual  STM  would  be  most  likely  to  lead  to  a 
picture  advantage. 

Method 

Subjects.  The  same  28  males  participated  in  this  experiment  as 
participated  in  Experiments  la,  2  and  3. 

Stimuli  and  Apparatus.  The  stimuli  for  the  STM  task  used  in 
this  experiment  were  the  same  six  words  and  six  pictures  as  used  in 
the  other  experiments  (see  Figure  1) .  The  test  displays  consisted  of 
pairs  of  pictures  or  words  as  described  in  Experiment  3 .  A  memory 
display  was  presented  preceding  the  test  display,  consisting  of 
three  to  five  of  the  picture  or  word  stimuli  arranged  in  either  a 
tabular  or  spatial  format  (see  Figures  6  and  7) .  From  a  viewing 
distance  of  43  cm,  the  tabular  display  subtended  a  visual  angle  of 
7.2  degrees  horizontally  and  12.5  degrees  vertically,  and  the 
spatial  display  subtended  a  visual  angle  of  14.2  degrees 
horizontally  and  vertically.  The  tabular  display  listed  each 
stimulus  in  a  separate  row,  accompanied  by  the  number  '  1'  or  '2', 
reflecting  that  item’s  priority.  The  spatial  display  consisted  of 


two  concentric  circles  with  the  pictures  or  words  spatially 
distributed  within  them.  A  stimulus  item  placed  within  the  inner 
circle  was  equivalent  to  a  priority  of  • 1  *  on  the  tabular  display, 
and  an  item  placed  within  the  outer,  circle  was  equivalent  to  a 
priority  of  '2'  on  the  tabular  display. 

The  apparatus  was  the  same  for  this  experiment  as  for  the 
previous  experiments.  The  1  z1  and  '/'  keys  were  again  used  to  make 
manual  responses,  and  as  in  Experiment  3,  they  were  labeled  'left' 
and  ' right 1 . 

Task  and  Procedure.  The  task  used  in  this  experiment  was 
similar  to  the  LTM  task  used  in  Experiment  3  in  that  subjects  were 
required  to  judge  which  item  of  a  pair  of  picture  or  word  test 
stimuli  was  the  greater  threat.  In  order  to  assure  that  the  task 
tapped  STM,  subjects  derived  threat  order  from  a  display  presented 
immediately  prior  to  the  test  stimuli  on  each  trial.  Memory  displays 
(see  Figures  6  and  7)  were  varied  in  three  ways.  They  were  either 
tabular  or  spatial  (memory  display  format) ,  contained  either  picture 
or  word  stimulus  items  (memory  display  stimulus  type) ,  and  contained 
three,  four  or  five  stimulus  items  (short-term  memory  load) . 
Additionally,  the  interval  between  the  onset  of  the  memory  display 
and  the  onset  of  the  test  display  was  varied. 

The  impetus  for  the  number  of  item  and  memory  interval 
manipulations  is  the  evidence  (e.g.,  see  Sperling  and  Speelman, 

1970)  that  the  visual  STM  decays  more  rapidly  and  retains  fewer 
chunks  of  information  than  the  auditory  STM.  Hence  when  only  a  few 
memory  items  are  stored  for  a  very  brief  time,  the  visual  STM  might 
be  used  to  store  these  items.  However,  as  the  number  of  items  stored 
and  storage  time  increase,  the  probability  is  greater  that  items 
will  be  coded  and  stored  in  the  auditory  STM.  The  assumption 
underlying  the  memory  display  stimulus  type  (picture/word)  and 
memory  display  type  (spatial/tabular)  manipulations  is  that 
textual/linguistic  material  would  be  more  likely  to  be  stored  in  the 
auditory  STM  and  spatial/pictorial  material  would  be  more  likely  to 
be  stored  in  the  visual  STM.  Which  STM  is  used  to  store  the  memory 
items  is  hypothesized  to  affect  the  speed  at  which  the  test  pictures 
and  words  can  be  processed:  Pictures  are  assumed  to  have  faster 
access  to  information  stored  in  the  visual  STM  and  words  faster 
access  to  information  stored  in  the  auditory  STM. 

Each  of  the  three  memory  loads  (three,  four  or  five  items)  was 
presented  as  pictures  and  words  on  spatial  and  tabular  memory 
displays,  resulting  in  12  possible  combinations  of  the  memory 
display  variables.  Two  memory  displays  were  generated  for  each  of 
these  combinations,  resulting  in  24  different  memory  displays.  Each  ’ 
memory  display  was  paired  with  eight  test  stimulus  pairs,  consisting 


of  four  different  pairs  presented  as  both  pictures  and  words.  This 
resulted  in  a  total  of  192  trials. 

Memory  display  stimulus  items  and  test  stimulus  items  were 
generated  such  that  each  stimulus  item  appeared  equally  often  as 
both  a  memory  and  test  item.  Stimulus  items  were  assigned  to 
specific  memory  displays  randomly,  with  the  restriction  that  the 
number  of  platforms  and  weapons  on  each  display  were  equal  for  four 
item  displays  or  differed  by  one  for  three  and  five  item  displays. 
Additionally,  whenever  possible,  one  weapon  and  one  platform  were 
given  the  same  priority  (tabular  display)  or  were  located  in  the 
same  circle  (spatial  display)  to  provide  a  basis  for  determining 
threat  order. 

The  test  stimulus  pairs  were  generated  randomly  from  the  items 
presented  on  the  appropriate  memory  displays,  with  the 
aforementioned  restriction  that  each  stimulus  item  appeared  an  equal 
number  of  times,  and  that  each  stimulus  item  was  the  greater  threat 
and  the  lesser  threat  of  the  pair  equally  often.  The  192  trials  were 
randomized  for  each  session. 

The  subjects  were  given  three  instructions  for  ordering  the 
stimulus  items  on  the  memory  display  according  to  threat  level. 
First,  items  within  the  inner  circle  on  the  spatial  display  and 
listed  as  priority  1 1 •  on  the  tabular  display  were  a  greater  threat 
than  items  in  the  outer  circle  or  listed  as  priority  '2'.  Second, 
for  priority  1 1 '  items,  weapons  were  a  greater  threat  than 
platforms.  Third,  for  priority  ’2*  items,  platforms  were  a  greater 
threat  than  weapons. 

The  interval  between  the  onset  of  the  memory  display  and  the 
onset  of  the  test  display  (short-term  memory  interval)  was  varied 
among  test  sessions  for  the  STM  task.  With  the  exception  of  the 
shortest  short-term  memory  interval,  the  memory  display  was  always 
presented  for  two  seconds;  For  the  shortest  short-term  memory 
interval,  the  memory  display  was  presented  for  700  msec.  Five 
short-term  memory  intervals  were  tested:  1.5  sec,  3  sec,  4  sec,  6 
sec,  and  8  sec.  Finally,  in  one  test  session,  trials  with  picture 
test  stimuli  were  separated  from  trials  with  word  test  stimuli  so 
that  trials  were  blocked  by  test  stimulus  type. 

The  procedure  for  this  task  was  basically  the  same  as  for  the 
tasks  used  in  the  other  experiments.  Each  practice  day  included  at 
least  one  practice  session  on  this  task.  Practice  and  tests  were 
performed  in  a  quiet,  darkened  room.  The  sessions  were  self-paced  in 
that  subjects  pressed  a  key  on  the  keyboard  to  begin  each  trial.  On 
practice  days,  the  short-term  memory  interval  for  the  STM  task  was 
set  at  five  seconds.  The  order  of  presentation  of  STM  task  test 


sessions  was  counterbalanced  across  subjects.  The  test  sessions  were 
no  different  than  the  practice  sessions,  except  for  the  short-term 
memory  interval  or  blocking. 

Results 


The  type  of  memory  display  (tabular  versus  spatial) ,  showed  no 
significant  main  effects  or  significant  interactions  in  any  of  the 
initial  analyses.  Hence  all  STM  task  data  were  re-analyzed  combining 
the  data  from  the  trials  using  tabular  and  spatial  memory  displays, 
and  these  latter  analyses  are  the  ones  reported. 

Several  different  analyses  were  computed  on  STM  task  data.  The 
STM  intervals  (time  from  onset  of  the  memory  display  to  the  onset  of 
the  test  stimuli)  were  different  for  the  first  14  and  second  14 
subjects.  Preliminary  analyses  of  the  first  14  subjects'  data 
indicated  that  STM  interval  did  not  have  the  hypothesized  effect  on 
the  processing  of  pictures  and  words.  I  decided  at  that  point  that 
the  intervals  tested  (3,  4  and  6  seconds)  might  not  sample  a  wide 
enough  range  of  intervals  to  show  the  hypothesized  effect,  so  the 
STM  intervals  for  the  second  14  subjects  were  set  at  1.5,  4,  and  8 
seconds.  The  first  14  subjects  were  also  given  a  4-second  STM 
interval  session  in  which  the  trials  were  blocked  so  that  subjects 
knew  what  type  of  memory  display  stimuli  and  test  stimuli,  that  is, 
pictures  or  words,  they  would  encounter  on  each  set  of  blocked 
trials.  The  preliminary  analysis  also  indicated  that  the 
blocking  variable  did  not  change  the  profile  of  results  across  the 
other  variables  of  interest  in  any  meaningful  way,  and  hence  the 
second  14  subjects  were  given  only  unblocked  trials.  Since  all  28 
subjects  were  tested  on  the  unblocked  4 -second  STM  interval  session, 
analysis  of  that  session  was  used  to  assess  the  other  variables  of 
interest.  Separate  analyses  (with  14  subjects)  were  used  to 
investigate  STM  interval  and  blocking  (blocked  versus  unblocked) . 

Analysis  with  all  subjects.  A  three-way  ANOVA  (Memory  Display 
Stimulus  Type  (MDST)  x  Test  Stimulus  Type  (TST)  x  Short-Term  Memory 
Load  (STML))  was  computed  on  all  28  subjects'  response  times  for  the 
unblocked  4-second  STM  interval  test  session.  Overall,  pictures  were 
responded  to  51  msec  faster  than  words  (598  msec  and  649  msec, 
respectively) (pc. 005) .  There  were  also  significant  response  time 
differences  among  the  three  short-term  memory  loads  (pc.OOl)  and 
several  two-  and  three-way  interactions.  Tests  of  simple  effects 
computed  on  the  two-way  interactions  showed  the  following:  (1)  MDST 
x  TST  interaction  (pc. 005)-  picture  test  stimuli  were  significantly 
faster  (pc.OOl)  than  word  test  stimuli  following  picture  memory 
display  stimuli,  but  only  marginally  (p=.07)  faster  than  word  test 
stimuli  following  word  memory  display  stimuli  (Figure  8);  and  (2). 
TST  x  STML  interaction  (pc. 05)-  picture  test  stimuli  were 


significantly  faster  than  words  for  STM  loads  of  three  (pc.OOl)  and 
five  (pc. 005),  but  there  was  no  difference  between  picture  and  word 
test  stimuli  for  a  STM  load  of  four  (Figure  9) . 

There  was  a  significant  (pc. 01)  MDST  x  TST  x  STML  interaction 
which  appears  to  result  from  the  fact  that  the  larger  advantage  for 
picture  test  stimuli  following  picture  memory  display  stimuli  than 
following  word  memory  display  stimuli  (the  source  of  the  MDST  x  TST 
interaction)  only  held  for  STM  loads  of  three  and  four.  With 
STM  loads  of  five,  the  picture  test  stimuli  advantage  was  eguivalent 
following  picture  and  word  memory  display  stimuli. 

Analyses  based  on  14  subjects.  The  first  14  subjects  performed 
four  test  sessions  of  the  STM  task.  The  three  'unblocked'  sessions 
had  STM  intervals  of  3,  4,  and  6  seconds.  The  'blocked'  session,  in 
which  subjects  knew  what  kind  of  memory  stimuli  and  test  stimuli 
they  would  be  presented,  had  a  STM  interval  of  four  seconds.  Two 
primary  analyses  were  computed  on  data  from  these  test  sessions:  A 
four-way  ANOVA  (STM  Interval  (STMI)  x  Memory  Display  Stimulus  Type 
(MDST)  x  Test  Stimulus  Type  (TST)  x  Short-Term  Memory  Load  (STML) ) 
was  computed  on  response  times  from  the  three  unblocked  sessions; 
and  a  four-way  ANOVA  (Block  (BLK)  x  MDST  x  TST  x  STML)  was  computed 
on  response  times  from  the  two  (blocked  and  unblocked)  4-second  STM 
interval  sessions. 

The  main  effects  and  interactions  involving  STM  interval  (STMI) 
were  the  primary  reasons  for  computing  the  first  analysis.  There 
were  no  significant  effects  involving  STMI.  The  significant  main 
effects  and  significant  interactions  not  involving  STMI  were  not 
explored  since  they  all  duplicated  those  found  and  discussed  in  the 
primary  analysis  of  the  unblocked  4-second  STMI  with  28  subjects. 

The  primary  concerns  of  the  second  analysis  were  the  main 
effects  and  interactions  involving  blocks.  Responses  to  the  blocked 
session  (575  msec)  were  significantly  faster  (F(l,13)  =  8.88,  pc. 05) 
than  responses  to  the  unblocked  session  (615  msec) .  There  was  also  a 
significant  Block  x  STM  load  interaction  (F(2,26)  =  3.81,  pc. 05). 
Tests  of  the  simple  effects  of  block  for  each  level  of  STM  load 
indicated  that  the  BLK  x  STML  interaction  was  due  to  the  fact  that 
responses  in  the  'blocked'  session  were  significantly  faster  than 
responses  in  the  'unblocked'  session  for  STM  loads  of  three  (pc.ooi) 
and  four  (pc. 05),  but  there  was  no  difference  between  response _ times 
for  the  'blocked'  and  'unblocked'  sessions  for  a  STM  load  of  five. 

Again,  all  significant  effects  not  involving  'block'  duplicated 
those  found  for  the  primary  analysis  of  28  subjects  in  the  unblocked 
4-second  STM  interval  session  and  will  not  be  further  explored  here. 


The  second  14  subjects  participated  in  three  test  sessions  of 
the  STM  task,  at  1.5-,  4-,  and  8 -second  STM  intervals.  A  four-way 
ANOVA  (STM  Interval  (STMI)  x  MDST  x  TST  x  STML)  was  computed  on 
response  times  for  the  three  test  sessions.  The  primary  concern  of 
this  analysis  was  the  effect  of  STM  interval.  There  was  a 
significant  difference  among  response  times  to  the  three  STM 
interval  sessions  (Pillais  =  .547,  Approx.  F(2,12)  =  7.26,  p<.01). 
Newman-Keuls  tests  computed  to  clarify  this  main  effect  showed  that 
response  times  for  the  1.5  STMI  session  were  significantly  faster 
(p<. 01)  than  response  times  for  either  of  the  other  STMI  sessions, 
which  were  not  different  from  each  other.  There  was  also  a 
significant  STMI  x  TST  x  STML  (F(4,52)  =2.58,  pc.05)  interaction. 
There  appears  to  be  no  simple  explanation  for  this  three-way 
interaction  involving  STMI,  but  it  does  not  seem  to  confound,  to  any 
serious  degree,  the  simpler  interactions  and  main  effects. 

All  main  effects  and  interactions  not  involving  STM  interval 
duplicated  those  found  in  the  primary  analysis  of  the  unblocked 
4-second  STM  interval  with  28  subjects. 

Discussion 


The  experimental  manipulations  for  the  STM  task  were  designed 
to  change  the  likelihood  of  memory  items  being  stored  in  visual  or 
auditory  STM.  The  hypothesis  underlying  these  STM  manipulations,  as 
stated  earlier,  is  that  fewer  memory  items,  shorter  memory 
intervals,  picture  memory  display  stimuli,  and  spatial  display 
formats,  are  more  likely  to  cause  memory  information  to  be  stored 
and  manipulated  in  a  visual  STM,  and  more  memory  items,  longer 
memory  intervals,  word  memory  display  stimuli,  and  textual  display 
formats  are  more  likely  to  cause  memory  information  to  be  stored  and 
manipulated  in  an  auditory  STM.  Further,  if  items  committed  to 
short-term  memory  are  stored  in  the  visual  STM,  a  task  which 
requires  access  to  that  information  would  be  performed  faster 
with  pictures  than  with  words.  Conversely,  if  items  committed  to 
short-term  memory  are  stored  in  the  auditory  STM,  then  a  task 
requiring  access  to  that  information  would  be  performed  faster  with 
words  than  with  pictures. 

The  overriding  result  for  the  STM  task  was  that  pictures  were 
responded  to  faster  than  words,  although  the  magnitude  of  this 
advantage  was  not  as  great  as  it  was  for  the  LTM  task.  This 
indicates  that  addition  of  an  STM  requirement  did  enhance  word 
processing  relative  to  picture  processing  compared  to  a  task  without 
the  STM  requirement,  but  that  a  significant  advantage  for  pictures 
remained  even  with  the  STM  requirement. 

Some  of  the  STM  manipulations  which  were  hypothesized  to  affect 
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the  probability  of  using  a  visual  or  verbal  STM  did  indeed  affect 
picture-word  response  time  differences,  but  even  for  the  conditions 
in  which  the  auditory  STM  should  have  been  most  likely  to  be  used, 
words  never  were  responded  to  faster  than  pictures.  There  were  four 
independent  variables  that  were  hypothesized  to  change  the 
probability  that  visual  or  auditory  STM  would  be  used  to  store  the 
memory  items  and  consequently  affect  picture-word  response  time 
differences.  These  were  STM  interval,  memory  display  stimulus  type, 
memory  display  format,  and  short-term  memory  load.  Additionally, 
block  (BLK)  was  analyzed  for  potential  clues  as  to  how  pictures  and 
words  are  processed  in  STM.  The  relevance  to  picture-word  processing 
of  the  results  involving  each  of  these  variables  will  be  discussed 
separately. 

Memory  display  format.  The  memory  display  format,  tabular  or 
spatial,  had  no  effect  on  picture-word  processing;  there  were  no 
significant  main  effects  or  interactions  involving  memory  display 
format  in  any  of  the  analyses.  Apparently,  the  background  or  context 
in  which  stimuli  are  presented  has  little  or  no  effect  on  how  those 
stimuli  are  memorially  coded  in  STM. 

Memory  display  stimulus  type.  Memory  display  stimulus  type 
(MDST)  and  short-term  memory  load  (STML)  were  primarily  assessed  in 
the  28-subject  analysis.  Both  variables  significantly  interacted 
with  test  stimulus  type,  that  is,  pictures  and  words.  Pictures  were 
always  responded  to  faster  than  words.  The  magnitude  of  the  picture 
advantage  was  less  following  word  memory  display  stimuli  than 
following  picture  memory  display  stimuli,  indicating  that  the  word 
memory  display  stimuli  did  enhance  test  word  processing,  but  not 
enough  to  offset  the  robust  response  time  advantage  for  pictures. 

Short-term  memory  load.  The  STML  x  TST  interaction  is  difficult 
to  interpret  on  first  inspection.  A  picture  advantage  exists  for 
short-term  memory  loads  of  three  and  five,  but  not  for  a  load  of 
four.  I  hypothesized  that  as  the  STML  increased,  the  picture 
advantage  should  decrease,  disappear,  or  even  be  reversed.  The 
negation  of  the  picture  advantage  when  the  STML  was  increased  from 
three  to  four  is  consistent  with  the  hypothesis,  but  the 
reappearance  of  a  picture  advantage  with  an  STML  of  five  is  not. 
However,  some  evidence  suggests  that  the  processing  that  occurred 
with  a  load  of  five  was  qualitatively  different  than  that  with  loads 
of  three  or  four.  Trends  in  the  data,  across  three  and  four  memory 
items  were  not  sustained  for  the  data  with  five  memory  items.  This 
is  reflected  in  the  BLK  x  STML  and  MDST  x  TST  x  STML  interactions, 
both  of  which  resulted  from  differences  between  the  trials  with  a 
STM  load  of  five  and  the  trials  with  STM  loads  of  three  or  four. 

These  changes  with  a  STM  load  of  five  and  anecdotal  reports  by 
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subjects  of  a  strategy  change  with  a  STM  load  of  five  suggest  that 
on  trials  with  five  memory  items  subjects  only  remembered  three  of 
them.  This  change  in  strategy  apparently  occurred  because  five  items 
in  this  experiment  approached  or  exceeded  STM  capacity.  When  five 
memory  items  were  presented,  subjects  apparently  committed  the  first 
three  threats  to  memory,  and  performed  the  task  as  usual  if  either 
member  of  the  test  pair  was  one  of  the  three  items  they  remembered. 
If  the  test  pair  included  both  items  (the  fourth  and  fifth  threat 
from  the  memory  display)  they  had  not  remembered,  then  they 
responded  to  the  platform  since  they  knew  the  platform  was  always 
the  greater  threat  of  the  last  two  in  the  threat  order.  Since 
different  strategies  seem  to  be  used  with  a  short-term  memory  load 
of  five,  depending  on  which  pair  was  presented  as  the  test  pair, 
this  condition  might  not  have  been  a  legitimate  test  of  the  STM  load 
of  five  condition. 

If  a  short-term  memory  load  of  five  is  accepted  as  a  special 
case,  then  the  change  in  picture-word  processing  across  STM  loads  of 
three  and  four  becomes  interpretable:  a  STM  load  of  four  enhances 
word  processing  relative  to  picture  processing.  This  appears  to 
mirror  the  effect  of  the  memory  display  stimulus  type,  that  is,  both 
word  memory  display  stimuli  and  a  larger  STM  load  reduce  the  picture 
advantage,  but  in  neither  case  is  a  word  advantage  produced. 

Short-term  memory  interval.  STM  interval  did  not  change 
picture-word  response  time  differences  in  any  systematic  way.  The 
hypothesis  was  that  as  the  STMI  became  longer,  visual  STM,  which  is 
presumably  shorter-lived  than  auditory  STM,  would  be  less  likely  to 
be  used  to  store  memory  items.  Although  there  was  one  higher  order 
interaction  involving  STMI,  it  did  not  indicate  any  systematic 
changes  compatible  with  the  hypothesis.  This  finding  suggests  that 
there  was  no  change  in  the  way  memory  items  were  stored  across  STM 
intervals  of  1.5  to  8  seconds.  There  were  absolute  response  time 
differences  due  to  STM  interval,  resulting  mainly  from  the  rapidity 
-  of  responses  in  the  shortest  (1.5  sec)  STM  interval  session.  This 
response  time  advantage  for  the  shortest  STM  interval  might  reflect 
a  classic  shift  in  speed-accuracy  strategy,  as  this  session  led  not 
only  to  faster  response  times  but  also  to  more  errors  than  the  other 
STM  interval  sessions.  The  shift  in  speed-accuracy  strategy  toward 
speed  for  the  1.5  second  STM  interval  might  have  been  induced  by  the 
increased  tempo  of  the  display  presentations,  since  both  the  memory 
display  presentation  time  and  the  interval  between  the  memory 
display  and  the  test  display  were  considerably  shorter  than  for  any 
other  STM  interval  session. 

Block.  Response  times  were  faster  overall  in  the  blocked 
session  than  in  the  unblocked  session.  This  indicated  that  there  was 
a  general  advantage  to  having  knowledge  about  the  type  of  stimuli 


that  would  be  presented  but  this  advantage  did  not  appear  to  change, 
in  any  meaningful  way,  the  difference  in  processing  pictures  and 
words.  One  explanation  of  the  picture  advantage  for  sessions  in 
which  subjects  did  not  know  whether  a  picture  or  word  would  be 
presented  on  each  trial,  is  that  subjects  used  a  strategy  favoring 
pictures,  that  is,  they  anticipated  picture  stimuli  and  so  were  more 
primed  for  pictures.  However,  if  this  were  the  case,  the  picture 
advantage  should  have  been  eliminated  for  the  blocked  session  since 
subjects  always  knew  whether  they  would  be  presented  with  pictures 
or  words,  and  therefore  they  could  have  anticipated  the  appropriate 
stimulus  type  in  all  cases. 

General  Discussion 

The  goal  of  this  series  of  experiments  was  to  determine  if 
faster  response  times  to  pictures  than  to  words  found  in  previous 
experiments  were  stimulus-  or  task-dependent.  The  finding  that  there 
was  a  picture  advantage  in  the  memory  tasks,  coupled  with  the 
absence  of  any  substantial  picture-word  response  time  differences  in 
the  recognition  tasks,  suggests  that  the  faster  response  times  to 
pictures  result  from  some  cognitive  processing  advantage  for 
pictures  in  general,  and  not  from  a  perceptual  processing  advantage 
for  the  visual  properties  of  a  particular  picture  set.  Hence  the 
response  -time  advantage  of  pictures  found  here  is  not  caused  by,  nor 
restricted  to,  a  particular  set  of  pictures. 

The  change  in  the  magnitude  of  the  response  time  advantage  for 
pictures  among  the  three  memory  tasks  used  in  this  study  indicates 
that  the  picture  advantage  is  affected  by  task  demands,  that  is,  the 
size  of  the  response  time  advantage  for  pictures  depends  on  the 
particular  task  involved.  However,  the  STM  task,  for  which  a  word 
response  time  advantage  was  predicted  on  the  basis  of  dual-coding 
theory,  led  instead,  to  a  picture  advantage.  Additionally,  task 
manipulations  made  within  the  STM  task  which  were  expected  to 
increase  the  likelihood  that  words  would  be  responded  to  faster, 
reduced  the  picture  advantage,  but  never  reversed  it.  Apparently, 
both  within  and  across  tasks,  there  are  variations  in  task  demands 
which  change  the  relative  efficiency  of  processing  pictures  and 
words,  but  overall,  there  is  an  advantage  in  the  processing  of 
pictures  which  cannot  be  easily  altered  by  biasing  the  task  heavily 
in  favor  of  verbal  processing. 

4 

In  the  absence  of  practical  constraints,  such  as  space  or 
computer  power,  it  seems  that  a  small  set  of  items  that  can  be 
easily  represented  by  simple  pictures  or  single  word  labels  should 
be  represented  by  pictures  regardless  of  the  task  demands.  Assuming 
that  there  are  no  visual  properties  of  the  picture  and  word  sets 
that  give  an  advantage  to  one  or  the  other  set  in  speed  of 
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recognition  (such  as  a  large  number  of  items  for  which  the  pictures 
are  difficult  to  discriminate  from  one  another) ,  the  rule  of  thumb 
to  use  pictures  appears  to  have  general  applicability. 


Footnotes 

1.  If  either  Bartlett’s  test  of  sphericity  or  the  Fmax  statistic  was 
significant,  then  the  multivariate  results  for  an  effect  are 
reported,  which  include  the  Pillais  value,  and  the  approximate  F  and 
degrees  of  freedom  based  on  multivariate  computations.  Otherwise, 
the  univariate  statistics  are  reported. 
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Figure  2.  Stimulus  Type  x  Item  Interaction  for  Recognition 
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Figure  4 .  Stimulus  Type  x  Item  Interaction  for  Recognition 

Task  with  Picture  Cues 


LU 


f- 

LU 

00 

o 

Cl 

cn 

LU 

CC 


STIMULUS  TYPE 


MINE  MISSILE  PLANE  SHIP  SUB  TORPEDO 


ITEM 


25 


0 


PLATFORM  WEAPON  ABOVE  BELOW  FAST 

CATEGORY 


SLOW 


Figure  6 .  Examples  of  Tabular  and  Spatial  Memory  Displays 

With  Word  Stimuli 
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Figure  8 .  Memory  Display  Stimuli  Type  x  Test  Stimuli  Type 

Interaction  for  STM  Task  (N=28) 
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